1. Field of the Invention
The invention relates to a system for and method of interactive, graphical computer simulation and, in particular, to a system and method for providing user programming of interactive, graphic and dynamic applications through pictorial means.
2. Description of the Related Art
Digital computers have long been used to perform complicated and tedious numeric calculations and data processing. With increased processing speed and memory capability, the roles of digital computers have expanded to include real-time modelling of mechanical, electrical, and other complex systems.
Definition. A computer graphics model visually resembles a product/system that is modelled, or represents the way a user thinks about a system without a direct visual analogy. A computer graphics model of a system or product may be used for developmental purposes (tradeoffs, human factors analysis, simulation), as the actual system/product operator interface, or for training purposes.
At first, software for performing digital computer modelling required advanced programming skills, the programmer coding the instructions in a low level language such as assembly language or machine code. This was done to provide the necessary interface between operator inputs and graphic objects displayed on an output device such as a graphic display or video monitor. With the development of high level languages, it became feasible to construct program generator systems to assist in the development of programs for performing the modelling. An example of such a modelling system is the Virtual Applications Prototyping System (VAPS) commercially available from the assignee of the subject application. The system is disclosed in system documentation publications including "VAPS: Conceptional Overview"; "VAPS: Reference Guide"; "VAPS: Programmers's Guide"; "VAPS by Example"; and "VAPS: Installation Guide and Options" all by Virtual Prototypes (1991) and incorporated herein by reference.
The VAPS system is a tool for designing, testing, and implementing user interfaces. A prototype or "modeled" system is the result of the design work done with design tools constituting the VAPS modelling system. A user interface is a part of hardware and software that provides for human control and feedback. For a software program, human control and feedback can include graphs, menus and prompts. For a product such as a radio, the interface can include scales, dials, and knobs. Thus, the user interface is a bridge for a human to access an application.
The resultant virtualized "prototype" system is a graphics model of a control and display mechanism which can be equivalent in appearance, form, functionality, behavior, animation, etc. to the modeled system and can also abstract parts of it. The virtualized prototype representation is also applicable to the generation of source code implementing equivalent functionality on a target system. Thus, as used herein, the prototype system is functionally equivalent to a modeled system. However, while the virtualized system is termed to be a prototype, it is not limited to initial system development but may constitute the final target system.
The VAPS system supports development and testing of dynamic, data driven graphical front ends for existing applications and maintenance of the user interface by allowing changes to be made visually. VAPS contains four primary modules or subsystems: (i) an Object Editor, (ii) an Integration Editor, (iii) a Logic Editor, and (iv) a Runtime Environment.
Briefly, the Object Editor is used by the operator to build the visual interface to the application. Using the Object Editor, the user draws objects that make up the graphical interface for the prototype. The objects are then assigned runtime behavior. The user employs the Integration Editor to define and build data connections or routes between the visual interface and an Application program data. That is, after the display image is produced using the Object Editor, the Integration Editor connects objects to each other, to data sinks, and to data sources. At the system level, data connections can be made by defining global prototype/applications system variables.
The Logic Editor allows the user to define and specify how events are to be managed. Using the Logic Editor, the user can develop a logic control program to manage events from the user interface. Finally, the Runtime Environment animates the prototype and allows the user to interact with the prototype.
In the VAPS system, the Object Editor is a window-based, graphical system used to draw and describe objects that make up a man-machine interface (MMI.) The VAPS MMIs are collections of objects having a visual representation created by the user and a functional behavior assigned to the graphical representation.
The Object Editor allows the user to build the "functional behavior" exhibited by complex objects or displays in runtime by specifying a list of object descriptions. The functional behavior of an object refers to how the graphic representation of an object responds to data received or generated by the prototype.
The object descriptions include two types of objects: input and output objects. In general, input objects react to user control to supply data to the simulation program representing the "prototype" or "modeled" system. Output objects react to data generated by the input objects, as well as to internal and external data sources of data. Output objects reflect the data by changing appearance in accordance with the predefined functional behavior of the respective object type. Thus, an output object such as a virtual meter might be graphically represented by a graduated scale and a pointer, the position of the pointer relative to the scale being responsive to predefined data generated by the prototype system.
Input objects accept user inputs and pass data to the system for processing. Typical classes of virtual input objects include buttons, locators, menus, knobs, switches, event labels, input fields, etc. Virtual input objects can be defined to be responsive to a corresponding physical input device, such as a mouse, keyboard, touch screen valuator, or a custom device. When a hardware system is modeled, the term "virtual object" as used in the specification and claims of this document is a graphical object that visually and functionally represents a real device usable for input/output purposes. When something other than a hardware system is modeled the term "virtual object" as used in the specification and claims of this document is a visual object with a well defined animation and/or behavior. The terms are used interchangeably in this document.
Once the application program has processed the information, the results are displayed on a video monitor screen using output objects. Typical classes of output objects include dials, lights, plots, cursers, bar charts, tapes, output fields, etc. Thus, output objects are different representations enabling viewing of data generated by the application. When the data changes, the output objects are dynamically modified to reflect the new value.
The functionality, i.e. behavior, of objects is defined using property sheets. The contents of a property sheet are dependent on the class of the object described by the property sheet. For example, a property sheet for a dial class (i.e., rotational) object defines the name of the object, display position origin, moving part, center of rotation, position pointer, initial pointer position, direction of motion, type of motion, and minimum and maximum positions and corresponding values.
As described, the Object Editor is used to draw the visual portion of the simulated prototype and define the functionality of objects. The Integration Editor Subsystem animates the objects by providing a data connection to the application data. Connections are made with the Integration Editor by specifying a correspondence or mapping between an object and a data source or data sink.
For example, if a "Dial" type of output object represents the temperature in a boiler, the temperature can be changed by an external source. The correspondence between the temperature and the Dial reading is made in the Integration Editor. Similarly, an input object that is created and functionally defined in the Object Editor is connected via globally defined data to provide user input to the application program.
Integration produces a data connection allowing virtual input objects to transmit data and output objects to receive data. The data controls the functional behavior of a virtual object or alters the attributes of a virtual object, such as a displayed position or shape of the virtual object.
Using the Integration Editor, data connections to and from a virtual object are made through a data channel. Data connections between virtual objects can be made from one virtual object to a series of other virtual objects. This is accomplished by storing the variable information from an input virtual object in a memory location, referred to as a data channel, and connecting the one or more output virtual objects to the same memory location to read the updated variable.
Data generated or consumed by data generators or consumers of the prototype interface are connected through direct point-and-click connections. Thus, an output virtual objects can directly receive data generated by the MMI. As used herein, "point-and-click" and "click on" refer to designation of an element or virtual object displayed on a video monitor. This is typically accomplished using a mouse, joystick, touch screen or other graphic input device to position a cursor displayed on the monitor (i.e., point) and activating a momentary switch to designate the element or virtual object at the cursor position (i.e., click or click-on) for further processing.
External user-defined application programs can also provide data. The memory locations for input and output virtual objects can be configured to transfer data to and from an external user application program, with the application program reading and writing to predefined data channels.
Finally, real devices can be directly integrated into the prototype with the interface. Such real devices include input devices such as sensors, joysticks, trackballs, and keyboards.
The Integration Editor maps user input and output information to data locations without regard to where the data originates. A pictorial interface maps data to provide a direct outlet to a virtual object called a plug. All virtual objects have such an associated plug, i.e., a labeled data location accessible under program control, to make the data available to virtual objects and applications programs. In comparison with having to program memory connections through a programming language, mapping using the Integration Editor allows the user to define how the virtual objects of the prototype are to communicate with each other by passing data back and forth using plugs.
Each mapping variable is defined by making a pictorial connection between the plug and the data source or sink associated with a virtual object. Therefore, when a virtual input objects of a prototype are connected to other data sources or sinks, only the information mapping needs to be changed, not the individual virtual object definition. Real devices can replace some or all the virtual objects, by being connected to the same data channel where a virtual object was connected, and inputting/outputting data in the same format.
Collectors are used to define required combinational Boolean or mathematical functions for application programs requiring the combination of several input sources to drive a single output virtual object. Collectors can also be used to determine a critical state by comparing an input value with a threshold value.
Data connections are also used to interface input objects of a prototype with an external application program. For example, interactive application programs may require control and display data interfaces to a prototype. The interface provides data communications between virtual objects and externally provided data. The development system also defines and provides communications with databases using connecting variables and remote or local communications.
The Logic Editor is used to model the prototype system by defining prototype system states and events. The Logic Editor uses a Finite State Machine (FSM) concept as the framework for MMI decision-logic development. This allows the prototype system to manage events in a context that takes system history into account and supports a fine-grained modelling of system functionality, potentially enhancing system reliability.
Prototype system behavioral modelling with the FSM concept produces a model containing a finite number of states. Each prototype state is entered and exited based on the occurrence of an event. Events can be user, environment or application generated.
To handle FSM modelling, the Logic Editor uses an Augmented Transition Network (ATN) language. The ATN language provides a structure for recognizing events and servicing transitions between prototype system states. Transitions are serviced by invoking action routines and then transitioning or moving to the next state.
The Logic Editor is used to produce an ATN program and define prototype system operation. The ATN language has two basic constructs, the STATE and the EVENT. These constructs map to the nodes and links of a conventional state diagram to define prototype interface decisions, i.e., provide logic control and action routines. The general structure of an ATN program is as follows:
______________________________________ STATE start.sub.-- state { initialize the state's parameters if required } ( EVENT event1) (condition) { response } .fwdarw. NEXT.sub.-- STATE ( EVENT event2) (condition) { response } .fwdarw. NEXT.sub.-- STATE ______________________________________
etc.
The design information captured by the Object Editor, Integration Editor and Logic Editor is available in a well-defined documented ASCII format, whereby for each object, this information includes all details regarding object class, object interfaces to controlling variables and object graphical primitives, attributes and transformations. The ATN network description also follows a well-defined grammar.
When a virtual event occurs, the prototype system is in a particular predefined state. The prototype system checks that the event that has occurred matches one of the transitions (i.e., an event and a condition) declared in that state. If the event is matched and the condition is true, i.e., valid and passed, the prototype system executes the response by executing a number of routines. These routines are programs, for example C subroutines. The prototype system then proceeds to the next state, awaiting other virtual events. Thus, the ATN code is used to define decision making of the prototype system.
Input events can be received by the ATN program and output virtual objects can be addressed by action routines invoked by the ATN program. The MMI can respond to events originating from many sources, including data entry, selection, discrete device input (such as a button, switch, input field, etc.), a continuous discrete device input reaching an increment (such as a potentiometer, knob or locator), time-out, periodic event, variable value crossing a threshold, voice input or a user-definable event.
When hardware is modelled, events can originate from real hardware devices, or "virtual hardware" (computer graphics replicas of similar look and functionality).
In order to service events, action routines are dispatched, i.e., invoked to service the event. The prototype development system makes libraries of action routines available to fulfill a variety of functions including performing specific object operations, manipulating graphical images and virtual objects on a virtual CRT screen, altering line types, patterns, fonts, and colors under program control, performing geometric transformations under program control, writing to output fields and reading from input fields, type checking for strings, communicating with an external simulation via Ethernet or other specialized buses, and remapping object plugs.
Once the functionality of the user interface is completed using the Object Editor, Integration Editor, and Logic Editor, the dynamics of the prototype are displayed at runtime in the Runtime Environment and the user is allowed to interact with the prototype through various real and simulated input devices. The Runtime Environments permits a user to evaluate the prototype and operation of the final MMI.
Using such a prototype development system, an essential part in the programming the human-computer interfaces for the prototype is to specify the behavior of the interface to virtual events. Such events may be generated by input devices (buttons, locators and triggers, keyboards, selects, function keys, touch screens, pictorial touch panels, voice interfaces) as well as by a timeout, a message from another system or a variable crossing a threshold value.
It is convenient to use a Finite State Machine (FSM) paradigm to respond to virtual events. Use of a FSM is suited to addressing sequencing problems often found in operator interfaces, and allows structuring of the control logic in an orderly, regular fashion.
When an FSM is used, events are processed in the context of a history of prior events and system states. A combination lock provides a good example of events interpreted in a context of prior events. Only the correct sequence will open the lock.
In another example, the pilot command "landing gear up" should only be serviced in the state "Weight-off-wheels" and not in the state "Weight-on-wheels". In a FSM, the system is deemed to be in one of a finite number of states. When input arrives, a transition is executed from the present state to the next state. The next state depends solely on the input and the present state. The output also depends solely on the input and the previous state. When applying the FSM model to operator interfaces, FSM input is made up of discrete events and FSM output is the dispatch of a series of subroutines or setting of plug variables. If a logical condition is tested before undertaking the transition, the FSM is an ATN.
ATNs or FSMs can be programmed with a text editor, by completing entries in a table or with a pictorial editor. However, textual completion of the table requires an in-depth knowledge of the syntax and structure of the ATN or FSM.
Accordingly, a need exists for an improved method and structure for defining the behavior of a system without requiring a knowledge of programming, FSM or ATN languages.
A further need exists for a method of, and structure for, defining a State Table using a graphical input and pull down, context sensitive menus.
A need further exists for a system to visually relate input and output virtual objects using a graphic input device.
A need further exists for a system which presents a user selection of valid relationships between virtual objects and available states.
A need further exists for a tool to specify a desired behavior using an graphic input device such as a mouse to point to, and designate, virtual objects which are part of the behavior and by designating desired virtual object behavior from menus.